M 


«)  Defense  wnem  ^ 

SmMGGMGNT  collggg 


PROGMM  l*MI'MG£M£NT  COURSE 
INDNIDIML  STUDY  PROGR/1M 


INDEPENDENT  RESEARCH  AND  DEVELOPMENT: 
THE  TECHNICAL  EVALUATION  PROCESS 

STUDY  PROJECT  REPORT 
PMC  77-2 

Earl  H.  Talley,  LtCol , USA 
and 

John  T.  Viola,  LtCol,  USAF 


D D C 

nVPREDP  n.'T’jf 

Uj  FEB  88  1978 

M^jEDUTSlk. 

^f=r  D 


fORT  DeLVIOIR,  VIIRCINW  02060 


DISTRIBUTION  STATEMENT  A 

Approved  for  public  release; 
Distribution  Unlimited 


REPORT  DOCUMENTATION  PAGE 

READ  INSTRUCTIONS 

BEFORE  COMPLETING  FORM 

1.  REPORT  NUMBER 

2.  GOVT  ACCESSION  NO. 

3.  RECIPIENT'S  CATALOG  NUMBER 

4.  TITLE  (and  Subtitle) 

INDEPENDENT  RESEARCH  AND  DEVELOPMENT  (IR&D) : 
THE  TECHNICAL  EVALUATION  PROCESS 

5.  TYPE  OF  REPORT  6 PERIOD  COVERED 

Study  Prolect  Report  77-2 

6.  PERFORMING  ORG.  REPORT  NUMBER 

7.  AUTHORS) 

EARL  H.  TALLEY 

JOHN  T.  VIOLA 

8.  CONTRACT  OR  GRANT  NUMBERf.) 

9.  PERFORMING  ORGANIZATION  NAME  AND  ADDRESS 

DEFENSE  SYSTEMS  MANAGEMENT  COLLEGE 

FT.  BELVOIR,  VA  22060 

10.  PROGRAM  ELEMENT,  PROJECT,  TASK 
AREA  6 WORK  UNIT  NUMBERS 

11.  CONTROLLING  OFFICE  NAME  ANO  ADDRESS 

DEFENSE  SYSTEMS  MANAGEMENT  COLLEGE 

FT.  BELVOIR.  VA  22060 

12.  REPORT  DATE 

77-2 

13.  NUMBER  OF  PAGES 

43 

14.  MONITORING  AGENCY  NAME  A ADDRESS(7f  different  from  Controlling  Office ) 

15.  SECURITY  CLASS,  (of  thta  report) 

UNCLASSIFIED 

1 I'lTT-l  II  1 —1 

16.  DISTRIBUTION  STATEMENT  (at  thle  Report) 


UNLIMITED 


DISTRIBUTION  STATEMENT  A 

Approved  for  public  release) 
Distribution  Unlimited 


17.  DISTRIBUTION  STATEMENT  (at  the  abstract  entered  In  Block  20,  It  dltlerent  from  Report) 


18.  SUPPLEMENTARY  NOTES 


19-  KEY  WORDS  (Continue  on  reverse  side  If  necessary  and  Identity  by  block  number) 


SEE  ATTACHED  SHEET 


20.  ABSTRACT  (Continue  on  reverae  aide  if  neceeeary  and  identify  by  block  numbar) 

SEE  ATTACHED  SHEET 


ItCtttIN  le 


ITII  Wiild  Section 

•06  lull  Section  O " 

IXIHIIO'JHCED  □ 

JUSTIFISATIUH 


IT 

CIS13I3UII3I1  tv, LIT V S.i'313 

Ditl,  AVAIL  sieJ  x 


DD  | jAN*73  7473  EDITION  OF  1 NOV  65  IS  OBSOLETE 


SECURITY  CLASSIFICATION  OF  THIS  PAGE  (When  Data  Entered) 


DEFENSE  SYSTEMS  MANAGEMENT  COLLEGE 


STUDY  TITLE: 


INDEPENDENT  RESEARCH  AND  DEVELOPMENT  ( IR&D) : 
EVALUATION  PROCESS 


THE  TECHNICAL 


STUDY  PROJECT  GOALS: 


To  become  familiar  with  the  mechanisms  of  the  IR&D  technical  evaluation  process 
as  it  is  intended  to  work;  to  determine  how  the  process  actually  works  in  prac- 
tice; and  to  identify  areas  of  improvement. 


STUDY  REPORT  ABSTRACT: 


The  purpose  of  this  report  is  to  present  the  results  of  an  analysis  and  evaluation 
of  the  IR&D  technical  evaluation  process.  DOD  and  military  department  documents 
and  regulations  were  examined  to  determine  implementation  procedures.  Interviews 
with  government  and  industry  IR&D  managers  are  summarized  and  evaluated  in  the 
report.  The  value  of  IR&D  is  presented  along  with  discussion  of  documentation 
from  public  law,  ASPR  and  DOD  policy  guidance  which  stipulates  requirements  for 
conducting  IR&D  evaluations.  Conclusions  and  recommendations  are  presented  based 
upon  an  analysis  of  the  information  gained  from  documentation  and  interviews. 


SUBJECT  DESCRIPTORS: 

Program/Project  Management  (10.02),  Major  Policies  (10.02.01) 
Procurement  Management  (10.07) 

Independent  Research  and  Development 


NAME,  RANK,  SERVICE 
Earl  H.  Talley,  LtCol , USA 
.John  T.  Viola,  LtCol , USAF 


CLASS 

PMC  77-2 


DATE 

November  1977 


INDEPENDENT  RESEARCH  AND  DEVELOPMENT: 
THE  TECHNICAL  EVALUATION  PROCESS 


Individual  Study  Program 
Study  Project  Report 
Prepared  as  a Formal  Report 


Defense  Systems  Management  College 
Program  Management  Course 
Class  77-2 


by 

Earl  H.  Talley 
LtCol  USA 

and 

John  T.  Viola 
LtCol  USAF 

November  1977 

Study  Project  Advisor 
Mr.  David  D.  Acker 


This  study  project  report  represents  the  views,  conclusions  and  recommenda- 
tions of  the  authors  and  does  not  necessarily  reflect  the  official  opinion 
of  the  Defense  Systems  Management  College  or  the  Department  of  Defense. 


EXECUTIVE  SUMMARY 


The  purposes  of  this  project  were  to  examine  the  Independent 
Research  and  Development  (IR&D)  technical  evaluation  process  as  it  is 
presently  managed  by  the  Office  of  the  Secretary  of  Defense  (OSD)  and 
the  three  military  departments;  to  evaluate  the  effectiveness  of  the 
process;  and  to  make  recommendations  for  improvement.  The  Department  of 
Defense  is  compelled  by  public  law  to  evaluate  IR&D  in  order  to  establish 
potential  military  relevancy  and  to  negotiate  advance  agreements  with 
selected  companies  to  establish  a dollar  ceiling  on  IR&D  costs.  Evalua- 
tions are  conducted  annually  on  a company's  IR&D  technical  plan  and  by 
means  of  on-site  reviews  every  three  years. 


Information  on  the  IR&D  technical  evaluation  process  was  obtained 
from  DOD  documentation  and  from  interviews  conducted  with  government  and 
industry  IR&D  managers.  The  authors  determined  that  the  Air  Force  pos- 
sesses the  most  highly  organized  evaluation  process  with  dedicated  per- 
sonnel assigned  the  sole  function  of  managing  the  IR&D  program.  The 
Army  and  Navy  assign  IR&D  evaluators  as  focal  points  on  an  additional 
duty  basis,  resulting  in  less  emphasis  on  the  evaluation  process. 

Our  findings  indicate  that  the  DOD  is  striving  to  improve  the  tech- 
niques and  methodology  of  the  evaluation  process.  Areas  the  authors 
identified  for  improvement  are:  identification  of  qualified  government 


personnel  to  participate  in  the  process  and  the  creation  of  a separate 
fund  citation  to  finance  the  cost  of  conducting  the  evaluations;  assur- 
ance of  continuity  in  the  evaluation  process  by  requiring  the  evaluation 
of  the  technical  plan  and  the  on-site  review  by  the  same  individual  and 


over  an  extended  period  of  time;  creation  of  a guide  tc  assist  evaluators 
in  conducting  a more  effective  evaluation;  and  the  creation  of  a document 


that  explains  IR&D  and  its  benefits  to  the  government  in  terms  of  its 
role  in  the  systems  acquisition  process. 

Industry  places  great  emphasis  on  IR&D  because  of  its  role  in 
improving  a firm's  competitive  standing.  Companies  value  direct  communica- 
tion and  interaction  between  their  own  managers,  scientists  and  engineers 
and  their  government  counterparts.  There  is  no  industry  consensus  on 
required  improvements;  however,  most  companies  believe  that  the  on-site 
review  is  more  beneficial  to  the  government  and  industry  than  the  evalua- 
tion of  the  IR&D  technical  plans. 
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SECTION  I 


INTRODUCTION 

Value  of  Independent  Research  and  Development 

The  continuing  close  relationship  between  U.  S.  defense  and  indus- 
trial R&D  communities  his  been  a major  contributor  to  the  technological 
leadership  on  which  our  national  security  and  economic  vitality  have 
depended  since  World  War  II.  The  Department  of  Defense  (DOD)  actively 
supports  and  draws  on  the  wide  base  of  advanced  technology  and  efficient 
production  processes  of  the  private  sector  for  the  superior  military 
hardware  essential  to  meet  national  security  requirements.  DOD  is  continu- 
ally seeking  methods  of  improving  the  ties  among  the  components  of  our 
national  R&D  community  and  to  strengthen  the  competitive  forces  on  which 
the  U.S.  depends  for  innovative,  efficient,  and  high-quality  military 
systems  ( 1 : 267 ) 1 . Independent  Research  and  Development  ( I R&D ) has  been 
recognized  as  an  important  component  of  this  process. 

IR&D  is  company-initiated  research  and  development  undertaken  to 
improve  a firm's  competitive  standing  by  virtue  of  improvements  in  pro- 
ducts, creation  of  new  products  or  advancement  of  the  company's  technolo- 
gical capability  in  general.  To  take  advantage  of  these  developments  DOD 
sponsors  IR&D  by  permitting  a closely  controlled  amount  of  the  relevant 
R&D  activities  undertaken  by  defense  contractors  to  be  recovered  as  an 
allowable  business  expense  on  their  government  contracts.  This  allowance 
is  small,  typically  one-to-three  percent  of  the  contract  value,  but  the 
payoffs  are  viewed  by  the  DOD  as  substantial. 

^ This  notation  will  be  used  throughout  the  report  for  sources  of  quotations 
and  major  references.  The  first  number  is  the  source  listed  in  the 
bibliography.  The  second  number  is  the  page  in  the  reference. 
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Support  of  IR&D  enhances  the  essential  element  of  competition  in 
the  development  and  procurement  processes  by  encouraging  contractors  to 
develop  their  ideas  and  capabilities  across  the  board.  Improved  contractor 
competence  can  mean  more  technology  gained  per  dollar  expended  and  can 
reduce  technical  risk.  In  addition,  a contractor,  anticipating  defense 
requirements,  may  develop  much  of  the  pertinent  technology  prior  to  formal 
DOD  commitment  to  a program,  thus  shortening  development  time. 

The  magnitude  of  the  IR&D  program  is  reflected  in  the  following  for 
contractor  fiscal  year  1976.  Industry  incurred  $1,232  billion  in  costs  for 
IR&D;  $1,061  billion  was  accepted  by  the  Government;  and  $0,543  billion  was 
recoverable  by  industry  as  overhead  in  DOD  contracts  (2).  Thus,  roughly  40 
percent  of  IR&D  costs  were  recoverable  from  DOD  contract  overhead. 

Despite  the  advantages  to  the  Government,  there  has  been  considerable 

controversy  relative  to  support  for  IR&D.  Criticism  was  expressed  by  some 

in  the  recent  past;  however,  this  criticism  has  somewhat  abated.  Perhaps, 

this  is  because  of  strong  support  for  IR&D  expressed  by  Dr.  Malcolm  Currie, 

former  Director  of  Research  and  Engineering. 

"IR&D  is  absolutely  central  to  the  quality  of  defense  RDT&E 
and  weapons  acquisition  and  I believe  that  its  'independence' 
must  be  maintained.  It  is  the  heart  of  a competitive  and 
competent  industrial  base:  it  results  in  lowering  the  cost 
of  acquisition  and  it  is  a uniquely  efficient  source  for  new 
technology... It  is  well  managed,  and  excellent  visibility 
is  provided  to  the  Congress.  It  pays  for  itself  many 
times  over.  I feel  that  further  controls  such  as  separate 
line  item  budget  approval  in  advance  by  Congress  would 
destroy  its  independent  and  innovative  character  and  be  a 
serious  loss."  (3:1-13). 

Congressional  leaders  have  also  expressed  support.  For  example. 
Senator  McIntyre  in  reviewing  the  "1976  DOD  Annual  Report  to  the  Congress 
on  IR&D  Costs"  stated: 
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"I  am  convinced  that  the  investment  we  make  in  independent 
research  and  development  is  not  only  prudent  but  essential. 

It  is  one  part  of  the  total  investment  to  preserve  our 
technological  leadership  and  is  one  that  has  paid  rich 
dividends." 

In  July  1977,  Senator  McIntyre  further  stated: 

"Mr.  President,  the  costs  for  IR&D  are  high  but  I believe 
that  the  present  system  strikes  a good  balance  between 
controlling  these  costs  and  giving  our  defense  industry 
sufficient  flexibility  to  maintain  a strong  technological 
base.  The  purpose  of  IR&D  funds  is  to  make  sure  there  are 
qualified  bidders  to  propose  on  DOD  programs.  It  is  the  price 
we  pay  to  make  sure  we  have  companies  that  are  on  the  fore- 
front of  technology  and  prepared  to  bid  on  new  projects. 

There  have  been  concerns  expressed  in  the  past  that  the 
Congress  does  not  have  adequate  control  of  IR&D  funds.  I 
do  not  share  this  view. 

It  is  clear  that  the  process  now  used  by  DOD  to  control 
the  amount  spent  for  IR&D  is  not  allowing  wild  increases,  in 
fact,  there  is  probably  a decline  in  real  value.  If  Congress 
were  to  become  more  involved  in  allocating  these  funds,  it 
would  nean  that  Congress  would  soon  have  to  deal  with  choices 
as  to  which  company  should  be  proficient  in  which  technology. 
Clearly,  those  decisions  must  be  left  to  the  individual 
companies.  In  my  judgment  the  present  system  strikes  a 
good  balance  between  control  and  flexibility." 


Purpose  and  Approach 

The  purpose  of  this  study  project  is  to  develop  an  understanding 
of  the  IR&D  technical  evaluation  process  in  order  to  evaluate  the 
effectiveness  of  the  process  and  to  recommend  improvement  where  necessary. 
Our  approach  was  to  examine  pertinent  literature  on  the  subject,  including 
appropriate  DOD  policy  and  procedural  documents,  and  to  collect  information 
by  means  of  structured  interviews  with  IR&D  managers  from  OSD,  military 
departments,  NASA  and  industry. 
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SECTION  II 


REQUIREMENTS,  POLICIES  AND  IMPLEMENTATION 
Requirements  for  IR&D  Evaluations 

The  requirements  for  technical  evaluations  of  company  IR&D  are  speci- 
fied by  Section  203,  Public  Law  91-441,  Appendix  C which  states  that 
"funds  authorized  for  appropriation  to  the  Department  of  Defense. . .shall 
not  be  available  after  December  31,  1970,  for  payment  of  independent  research 
and  development  or  bid  and  proposal  costs  unless  the  work  for  which  payment 
is  made  has,  in  the  opinion  of  the  Secretary  of  Defense,  a potential  rela- 
tionship to  a military  function  or  operation..."  (5) 

The  law  further  stipulates  that  the  Secretary  of  Derense,  prior  to  or 
during  each  fiscal  year,  must  negotiate  advance  agreements  establishing  a 
dollar  ceiling  on  such  costs  with  all  companies  which  during  their  last  pre- 
ceding fiscal  year  received  more  than  $2  million  of  IR&D  or  B&P  payments  from 
the  DOD.  The  IR&D  portions  of  the  advance  agreements  thus  negotiated  must 
be  based  on  company  submitted  plans  on  each  of  which  a technical  evaluation 
is  performed  by  the  DOD  prior  to  or  during  the  fiscal  year  covered  by  such 
advanced  agreements. 

The  DOD  has  undertaken  to  comply  with  these  requirements  in  an  efficient 
and  effective  manner. 

Department  of  Defense  policy 

Department  of  Defense  policies  regarding  IR&D  and  provisions  for 
complying  with  Section  203  of  PL  91-441  are  set  forth  in  DODI  5100.66, 
dated  January  7,  1975  (6).  It  defines  the  IR&D  Policy  Council  as  an  organiza- 
tion composed  of  Assistant  Secretaries  of  Defense  and  Assistant  Secretaries 
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of  the  military  departments  and  chaired  by  the  Director  of  Defense  Research 
and  Engineering  (DDR&E).  This  Council  is  charged  with  developing  and 
disseminating  DOD  policy  and  guidance  essential  to  the  administration  of 
the  DOD  IR&D  program.  It  stipulates  that  relevancy  determination  is  part 
of  the  evaluation  process  and  identifies  one  of  the  military  departments  as  a 
Lead  Department  in  each  evaluation.  The  department  is  then  responsible  for 
arranging  and  conducting  on-site  reviews  and  for  coordinating  and  summarizing 
technical  evaluations  of  project  descriptions  in  a contractor's  IR&D  technical 
plan. 

This  Instruction  expresses  the  principle  that  DOD  recognizes  IR&D  as  a 
necessary  cost  of  doing  business  and  seeks  to  encourage  the  creation  of  an 
environment  which  enhances  development  of  innovative  concepts  for  defense 
systems  and  equipment.  It  strives  to  develop  technical  competence  in  two 
or  more  contractors  who  can  respond  competitively  to  requirements  sought  by 
DOD  from  industry.  It  states  that  the  basic  purpose  of  the  IR&D  technical 
evaluation  is  to  assist  in  the  determination  of  the  potential  military 
relationship  and  to  assist  in  the  evaluation  of  the  reasonableness  and 
technical  quality  of  the  contractor  IR&D  program. 

T'-is  Instruction  established  the  computer-based  IR&D  data  bank  at  the 
Defense  Documentation  Center  to  aid  in  the  coordination  of  the  DOD  contract 
R&D  and  in-house  R&D  programs  with  the  IR&D  programs.  It  was  designed  to  be 
a centralized  body  of  information  which  is  useful  in  identifying  IR&D 
projects. 

Tri-Service  participation  in  technical  plan  evaluation,  on-site  review 
and  advance  agreement  prenegotiation  is  strongly  encouraged  to  foster 
technical  interchange  and  uniform  treatment  of  contractors  by  DOD  Components. 

As  stated  in  the  Instruction,  the  important  objectives  of  the  on-site 
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review  are  to  permit  face-to-face  technical  dialogue  between  government  and 
industry  peers  and  to  confirm  the  technical  plan  evaluation  rating. 

The  DOD  policy  assigns  the  Secretaries  of  the  military  departments  the 
responsibility  of  evaluating  the  project  descriptions  and  conducting  the 
on-site  review  of  projects  at  each  company  at  least  once  every  three  years. 

The  IR&D  Technical  Evaluation  Group  is  composed  of  the  three  military 
department  IR&D  managers  and  a chairman  appointed  by  the  DDR&E.  The  Technical 
Evaluation  Group  is  responsible  for  establishing  methodology,  criteria, 
and  procedures  for  performing  the  technical  evaluation  of  company  IR&D 
projects. 

Each  Departmental  IR&D  Manager  designates  the  organizations  within 
his  military  department  that  are  responsible  for  IR&D  evaluation.  He  ensures 
effective  evaluation  of  IR&D  project  descriptions;  arranges  for,  and  partici- 
pates in,  on-site  reviews;  and  maintains  an  up-to-date  distribution  list 
for  IR&D  Technical  Plans  (also  known  as  brochures). 

Armed  Services  Procurement  Regulation 

Paragraph  15-205.35,  Section  XV  of  ASPR  establishes  cost  principles 
and  procedures  regarding  IR&D  negotiations.  It  states  that  "a  contractor's 
IR&D  effort  is  that  technical  effort  which  is  not  sponsored  by,  or  required 
in  performance  of,  a contract  or  grant  and  which  consists  of  projects 
falling  within  the  following  three  areas:  (i)  basic  and  applied  research, 
(ii)  development,  and  (iii)  systems  and  other  concept  formulation  studies. 
IR&D  effort  shall  not  include  technical  effort  expended  in  the  development 
and  preparation  of  technical  data  specifically  to  support  the  submission  of 
a bid  and  proposal ."  (7) 
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The  ASPR  provides  that  any  company  that  received  payments  as  prime  or 
sub-contractor,  in  excess  of  $2  million  from  the  DOD  for  IR&D  and  B&P  in  a 
fiscal  year,  is  required  to  negotiate  an  advance  agreement  with  the  government 
which  establishes  a ceiling  for  allowability  of  IR&D  costs  for  the  following 
fiscal  year. 

It  states  that  in  negotiating  a ceiling  particular  attention  must  be 
paid  to  such  factors  as  the  technical  evaluation,  including  the  potential 
relationships  of  IR&D  projects  to  a military  function  or  operation,  and 
comparison  with  the  previous  years'  program  including  the  level  of  the  govern- 
ment's participation. 

It  is  important  to  note  that  DOD  does  not  make  cash  payments  to  the  con- 
tractor for  allowable  IR&D  costs.  These  costs  must  be  recovered  as  overhead 
which  is  included  as  part  of  the  price  of  products  sold  to  DOD  in  the  year 
following  the  negotiation  of  an  agreement.  If  a contractor  does  no  defense 
work  during  that  year,  the  DOD  incurs  no  cost.  The  costs  related  to  products 
and  services  that  are  sold  to  the  government  are  recoverable  up  to  the  dollar 
ceiling  previously  negotiated. 

Common  Areas  of  Implementation 

DODI  5100.06  seeks  to  involve  Tri-Service  participation  in  the  evaluation 
of  a company's  IR&D  program.  Therefore,  common  areas  exist  for  the  implementa- 
tion of  DOD  policies  on  the  technical  evaluations.  These  common  areas 
include  criteria  for  the  nature  and  types  of  tasks,  requirements  for  the 
negotiations  of  funding  limits,  establishment  of  IR&D  relevancy  in  areas  in 
which  DOD  needs  competition,  review  of  the  IR&D  tasks  as  planned  and  then 
review  when  complete,  and  acceptance  or  rejection  of  IR&D  costs  based  on 
adherence  to  its  criteria. 


ASPR  requires  companies  to  submit  their  IR&D  Technical  Plans  for  evalu- 
ation by  service  and  NASA  representatives.  On-site  reviews  are  commonly 
attended  by  these  representatives  and  technical  evaluations  are  scored  on 
the  DO  Form  1855,  "Independent  Research  and  Development  Project  Technical 
Evaluation".  Evaluation  results  on  specific  projects  are  tabulated  on  DD 
Form  1856,  "IR&D  Project  Technical  Evaluation  Summary".  These  completed 
forms,  together  with  written  summaries,  are  used  in  the  negotiation  process. 

Army  Implementation 

Responsibilities  and  procedures  for  the  technical  evaluation  of  IR&D 
programs  of  DOD  contractors  by  the  Army  are  contained  in  Army  Materiel 
Command  Regulation  70-40. 

The  Assistant  Deputy  for  Science  and  Technology,  Headquarters  DARCOM, 
is  responsible  for  carrying  out  the  Army's  mission  in  the  DOD  IR&D  program. 
His  agent  in  this  activity  is  the  Departmental  IR&D  Manager,  the  Army's 
representative  on  the  IR&D  Technical  Evaluation  Group.  The  commander  of 
each  subordinate  command  and  the  director  of  each  laboratory  are  responsible 
for  evaluating  the  Technical  IR&D  Plans,  designating  IR&D  points  of  contact 
within  their  organizations  and  providing  personnel  to  conduct  on-site 
reviews  (8). 


Navy  Implementation 

Secretary  of  Navy  Instruction  3900.40  assigns  responsibilities  and 
outlines  procedures  for  the  conduct  of  IR&D  project  description  evaluations 
and  on-site  reviews  of  IR&D  programs  within  the  Navy  (9).  This  StXNAV 
Instruction  is  implemented  by  Office  of  Naval  Research  (0NR)  Instruction 
3900.32  and  Naval  Materiel  Command  Instruction  3900. 11 A (10  and  11). 
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The  Chief  of  Naval  Research  is  designated  as  the  Navy  IR&D  Program 
Manager.  He  is  responsible  for  the  overall  conduct  of  IR&D  project  descrip- 
tion evaluations  and  on-site  reviews  of  the  IR&D  programs. 

As  outlined  in  the  above  ONR  implementing  instruction,  the  Director  of 
Research,  ONR,  is  designated  as  the  Navy  member  of  the  DOD  IR&D  Technical 
Evaluation  Group.  His  special  assistant  serves  as  alternate  member  of 
the  group  and  as  the  overall  Navy  focal  point  for  coordination  of  all  IR&D 
actions,  to  include  technical  plans  and  on-site  reviews. 

Each  of  the  ONR  Branch  Offices  serves  as  Lead  Navy  Activity  for  designated 
companies  and  conducts  technical  evaluations  of  their  IR&D  project  descriptions 
and  on-site  reviews. 


Air  Force  Implementation 

As  indicated  in  AF  Regulation  80-63,  the  basic  policy  of  the  Air  Force 
is  that  it  supports  and  encourages  the  concept  of  contractor  IR&D  as  a 
vital  and  necessary  means  of  stimulating  scientific,  technological,  and 
development  efforts  on  the  part  of  DOD  contractors.  Contractors'  efforts 
should  remain  independent  of  Air  Force  control  except  for  a technical 
evaluation  of  their  quality  and  their  potential  relationship  to  a military 
function  or  operation.  The  results  of  technical  evaluations  are  used  in 
the  determination  of  negotiation  objectives  for  IR&D  advance  agreements  (12). 

In  addition  to  basic  guidance  by  DODI  5100.66,  the  Air  Force  has 
issued  a number  of  regulations  and  operating  instructions  that  are  designed 
to  aid  in  the  IR&D  technical  evaluation  process.  AFR  80-17,  "Air  Force 
IR&D  Policy  Council",  specifies  the  purpose  and  composition  of  the  Policy 
Council.  Members  include  ASAF/RD&L,  Chairman,  and  representatives  from 
AFSC  and  the  Air  Staff.  The  Executive  Secretary  is  appointed  by  the  chairman. 
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The  Air  Force  IR&D  Technical  Program  Manager  and  the  Air  Force  Tri-Service 
Negotiation  Chairman  are  advisors  to  the  Council.  The  Council  is  the  Air 
Force  focal  point  for  establishing  overall  IR&D  policy,  and  serving  as  the 
interface  with  the  DOD  IR&D  Policy  Council  (13). 

AFR  80-53,  "Technical  Evaluation  of  IR&D",  states  policy,  assigns 
responsibilities,  and  describes  procedures  for  making  technical  evaluations 
and  reviews  of  contractor  IR&D  programs.  It  designates  DCS/  Research  and 
Development,  HQ  USAF,  as  the  office  of  primary  responsibility  for  IR&D 
technical  matters.  The  Commander,  AFSC,  is  responsible  for  the  administration 
and  management  of  the  IR&D  program  and  for  implementation  of  policy  guidance 
and  for  the  designation  of  the  Air  Force  IR&D  Technical  Manager.  The 
Technical  Manager  is  responsible  for  evaluating  written  descriptions  of 
IR&D  projects,  for  conducting  on-site  technical  reviews  of  projects  and 
company  IR&D  programs  and  for  preparing  written  evaluation  reports  of  these 
reviews.  When  the  Air  Force  is  the  lead  department,  the  Technical  Manager 
must  verify  that  the  overall  evaluation  has  covered  at  least  90  percent  of 
the  dollar  value  of  each  company's  IR&D  program  to  ensure  that  the  evaluation 
is  valid. 

The  Technical  Manager  represents  the  Air  Force  on  the  DOD  IR&D  Technical 
Evaluation  Group.  Among  other  responsibilities,  he  designates  the  AFSC 
lead  organization  for  each  company  division  for  which  the  Air  Force  is  lead 
department  and  monitors,  reviews,  evaluates,  and  guides  the  Air  Force 
technical  evaluation  of  contractors  IR&D  programs. 

AFSC  Supplement  1 to  AFR  80-53  provides  further  guidance  and  instruc- 
tion on  conducting  the  evaluations.  It  specifically  states  that  each  AFSC 
Commander  is  to  actively  support  the  IR&D  technical  evaluation  process  and 
ensure  that  technical  evaluations  that  fall  within  his  command  are  accomplished 
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by  highly  qualified  personnel  and  with  dispatch.  It  provides  that  the  HQ 
AFSC  Director  of  Science  and  Technology  will  designate  a focal  point  for 
IR&D  matters  and  ensure  that  projects  are  evaluated  by  well -qualified 
personnel.  The  focal  points'  organization  will  serve  as  lead  for  a particular 
company  evaluation  and  discharge  assigned  duties.  The  AFSC  supplement  further 
specifies  that  at  least  30  percent  of  the  contractor's  IR&D  program  should 
be  reviewed  at  the  on-site  review  and  indicates  certain  standards  regarding 
evaluator  qualifications  and  coverage  by  different  organizations  and  depart- 
ments to  ensure  a broad  based  evaluation  (14). 

The  Air  Force  Tri-Service  Negotiation  Group  in  DCS  Procurement  and 
Manufacturing  of  HQ  AFSC  is  responsible  for  negotiating  advance  agreements 
in  accordance  with  PL  91-441  and  ASPR  Section  15,  Part  2 for  companies 
under  Air  Force  cognizance.  It  uses  the  technical  ratings  and  determinations 
of  Potential  Military  Relevance  (PMR)  prepared  by  the  Air  Force  Technical 
Manager  to  establish  the  IR&D  ceilings.  Past  experience  has  shown  that 
over  90  percent  of  the  IR&D  and  B&P  projects  have  PMR.  (15) 

In  1977,  the  total  proposed  IR&D  program  for  companies  under  Air  Force 
cognizance  is  $983  million.  Air  Force  Tri -Service  negotiated  a total  IR&D 
ceiling  of  $747  million  with  39  companies.  This  included  55  agreements  and 
120  separate  ceilings.  The  DOD  allocatable  share  is  $314  million.  The 
remaining  ceiling  of  $433  million  is  allocated  to  commercial  and  other 
government  contracts. 

NASA  IR&D  Implementation  and  Participation 

NASA  and  DOD  share  common  objectives  in  investigations  of  many  areas 
of  technology,  therefore,  it  has  been  and  continues  to  be  the  practice  of 
NASA  to  participate  in  the  technical  evaluation  and  negotiation  of  IR&D 
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programs  of  companies  with  whom  NASA  has  a substantial  business  and  techno- 
logical interest.  It  is  NASA  policy  to  evaluate  selected  contractor  IR&D 
Technical  Plans  and  provide  the  results  to  the  cognizant  DOD  organization 
and  to  provide  adequate  support  to  the  DOD  in  conducting  on-site  reviews. 
NASA  representatives  are  identified  to  serve  on  the  IR&D  Policy  Council  and 
on  the  DOD  IR&D  Technical  Evaluation  Group. 

The  directors  of  NASA  field  installations  are  tasked  to  designate  a 
focal  point  for  IR&D  technical  evaluations,  to  participate  in  on-site 
reviews,  to  interact  directly  with  contractor  IR&D  managers,  to  conduct 
program  assessments  and  to  identify  NASA  technology  needs  to  industry. 
Significant  policy  guidance  is  indicated  in  the  NASA  Management  Instruction 
regarding  IR&D  (16). 

NASA  participation  in  IR&D  has  increased  in  recent  years  despite  the 
fact  that  they  are  not  required  to  do  so  by  Public  Law.  For  example,  in 
1975  NASA  allowed  $40  million  of  IR&D  in  advance  agreements  signed  by  DOD 
negotiators. 


SECTION  III 


DISCUSSION  AND  ANALYSIS  OF  INTERVIEWS  WITH  IR&D  MANAGERS 

General  Comments 

Research  activities  that  were  conducted  for  the  collection  of  data  and 
information  involved,  primarily,  interviews  with  DOD,  NASA  and  industry 
officials  directly  involved  with  the  conduct  and  evaluation  of  IR&D.  The 
authors  attempted  to  interview  each  of  the  officials  personally;  however, 
in  one  case  a telephonic  discussion  was  necessary.  Each  of  the  interviews 
involved  the  questions  identified  below.  Attempts  were  made  to  speak  with 
individuals  who  have  had  a continuing  involvement  with  IR&D  over  the  years 
so  as  to  gain  an  understanding  nf  the  evaluation  of  DOD-Industry  interactions 
in  this  area.  A list  of  persons  interviewed  is  found  in  Appendix  A. 

Basic  Questions  for  Interviews 

In  order  to  structure  the  interviews  the  following  questions  served  to 
initiate  discussion  and  to  highlight  areas  of  primary  interest: 

1.  How  is  the  IR&D  technical  evaluation  process  supposed  to  work? 

2.  How  does  it  actually  work? 

3.  Is  this  method  satisfactory?  If  not,  what  are  some  considerations 
that  might  improve  the  process? 

4.  Can  you  suggest  other  references  or  sources  of  information? 

In  actual  fact,  many  other  questions  were  addressed;  therefore,  the 
report  of  the  information  gathered  during  the  interviews  is,  in  some  cases, 
not  structured  according  to  the  format  of  these  questions.  The  authors 
have  attempted  to  present  in  each  case,  the  information  in  terms  of 
how  the  interviewee  perceives  the  IR&D  technical  evaluation  process. 


Specific  Information  Obtained  From  Interviews 


Office  of  the  Secretary  of  Defense 

With  the  exception  of  policy  statements  provided  to  Congress  by  ODDR&E, 
there  has  been  little  recent  direct  involvement  at  the  OSD  level  in  IR&D 
policy  matters.  For  example,  the  last  meeting  of  the  IR&D  Policy  Council 
occurred  in  August  1975  in  order  to  address  specific  comments  on  IR&D 
published  in  a GAO  report.  During  the  interim  OSD  has  continued  to  manage 
by  exception. 

It  was  felt  that  OSD  leaders  appointed  by  the  new  administration  would 
continue  the  same  policies  on  IR&D  that  were  expressed  by  Dr.  Currie  and 
Dr.  Foster.  The  IR&D  Policy  Council  and/or  Technical  Evaluation  Group  should 
meet  shortly  after  the  current  OSD  reorganization  is  completed  and  reveal  new 
faces  and  policies,  if  any. 

It  was  indicated  that  although  the  Department  of  Energy  (DOE)  does 
not  yet  participate  in  the  technical  evaluations,  they  may  soon  begin  to  do 
so  if  relevancy  issues  are  resolved.  Several  companies  have  described  in 
their  technical  plans  technology  areas  that  appear  to  be  quite  relevant  to 
DOE  interests. 

The  OSD  official  pointed  out  that  companies  maintain  and  develop  com- 
petence in  DOD  relevant  areas  with  the  aid  of  IR&D.  This  fosters  the 
growth  of  competition  which  is  a necessary  part  of  cost-effective  procure- 
ment. One  could  assume  that  IR&D  quality  and  relevancy  would  be  commensurate 
with  DOD  needs  because  companies  will  naturally  seek  to  improve  their 
competitive  position  vis-a-vis  potential  DOD  contract  awards.  Nevertheless, 
technical  evaluation  and  determination  of  potential  military  relevancy  of 
IR&D  is  required  by  Public  Law. 


The  Chairman  of  the  Technical  Evaluation  Group  believes  that  an 
important  feature  of  the  technical  evaluation  is  the  opportunity  it  provides 
to  the  industry  IR&D  team  to  review  their  activities  and  to  prepare  descrip- 
tive technical  plans  which  are  made  available  to  DOD  scientists.  An  important 
benefit  of  this  is  seen  to  be  the  fostering  of  technical  interchange,  a 
process  which  may  not  occur  in  the  absence  of  the  IR&D  technical  evaluations. 
The  principal  problem  with  the  process  is  the  uncertainty  of  whether  or  not 
the  correct  DOD  member  is  participating  in  the  evaluation  and  involved  in  the 
technical  interchange.  OSD  has  attempted  to  derive  a scheme  whereby  the 
IR&D  Data  Bank  at  DDC  can  be  used  in  conjunction  with  DD  Form  1498 ' s for 
DOD  R&D  activity  to  matchup  common  R&D  tasks. 

All  IR&D  task  descriptive  information  is  stored  in  the  data  bank  by 
submission  of  the  Form  271  which  is  contained  on  the  front  of  each  Task 
Plan  when  submitted  by  the  company.  Use  of  COSATI  codes  or  key  words  has 
not  successfully  and  unambiguously  provided  the  matchup  because  the  codes 
are  too  general  and  key  words  are  too  numerous. 

The  identification  of  the  correct  DOD  scientist  would  facilitate  more 
effective  technical  evaluation  and  interchange.  OSD  is  continuing  to  work 
this  problem. 

IR&D  requires  additional  explanation  especially  to  the  acquisition 
command  member  such  as  program  managers  because  some  in  the  past  have  been 
reluctant  to  pay  for  this  as  overhead  in  their  programs. 

Stimulus  of  laboratory  personnel  is  also  recommended  in  order  to 
involve  key  personnel  in  technical  interchange. 

Army 

How  is  the  IR&D  Technical  Evaluation  Process  Supposed  to  Work? 
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All  personnel  interviewed  in  the  Army  were  thoroughly  familiar  with 
the  technical  evaluation  process  as  contained  in  appropriate  DOD  instruc- 
tions and  Army  regulations. 

Annually  the  Army  as  lead  service  evaluates  approximately  21  technical 
plans  and  conducts  approximately  seven  on-site  reviews. 

How  Does  the  Process  Actually  Work? 

The  regulation  mentioned  in  Section  II  above  fully  implements  the 
policies  contained  in  DOD  Instruction  5100.66.  Primary  responsibility  for 
IR&D  has  been  delegated  to  the  Commanding  General,  DARCOM.  Representatives 
in  the  Secretary  of  the  Army's  office  and  on  the  Department  of  the  Army 
staff  serve  as  focal  points  at  their  respective  levels,  monitor  IR&D  actions, 
and  keep  the  Secretary  of  the  Army  and  Chief  of  Staff  informed  as  necessary. 

The  key  figure  in  the  IR&D  process  is  the  Army  Departmental  IR&D 
Manager.  In  addition  to  assisting  the  Assistant  Deputy  for  Science  and 
Technology  in  carrying  out  the  Army’s  mission  in  the  DOD  IR&D  program,  he 
is  responsible  for: 

• Representing  the  Army  on  the  IR&D  Technical  Evaluation  Group. 

• Designating  appropriate  subordinate  commands  to  conduct  technical 
evaluations  of  the  IR&D  Technical  Plans  that  are  submitted  to  the  Army. 

• Coordinating  and  participating  in  on-site  reviews. 

• Assuring  the  maintenance  of  an  up-to-date  distribution  list 
for  IR&D  Technical  Plans. 

• Coordinating  Army  evaluation  of  appropriate  IR&D  that  is 
sponsored  by  other  services. 

• Furnishing  the  procurement  office  with  copies  of  the  evalua- 
tions for  companies  conducting  Army- sponsored  IR&D. 
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In  addition  to  providing  necessary  support  to  the  IR&D  Departmental 
Manager,  each  DARCOM  subordinate  commander  designates  an  individual  as 
point-of-contact  for  his  organization. 

Interviews  with  industry  officials  reveal  that  the  Army's  system  of 
conducting  on-site  reviews  and  particularly  providing  respective  companies 
with  "feedback"  after  technical  evaluations  is  more  than  adequate. 

Is  This  Method  Satisfactory?  If  Not,  What  Are  Some  Considerations  That 
Might  Improve  the  Process? 

The  personnel  interviewed  feel  that  the  IR&D  technical  evaluation 
process  is  an  important  and  necessary  function.  It  provides  a useful 
interface  between  technical  personnel  in  industry  and  in  the  Services. 

However,  the  effective  implementation  of  the  program  is  hampered  by  budget 
and  personnel  constraints.  Sufficient  funds  are  not  always  available  for 
the  proper  number  of  qualif.ed  personnel  to  participate  in  on-site  reviews. 
Current  personnel  manning  levels  determine  the  amount  of  effort  that  a 
given  office  can  devote  to  this  subject.  The  Army  does  not  have  personnel 
at  any  level  who  are  solely  dedicated  to  IR&D.  As  an  example,  the  IR&D 
Departmental  Manager  is  the  Chief  of  the  Industry  Liaison  Office  in  Headquarters, 
DARCOM.  IR&D  is  just  one  of  his  many  responsibilities. 

Similar  to  any  other  process,  command  backing  is  a necessary  ingredient. 

With  proper  backing  and  adequate  personnel  and  administrative  funds  to 
support  the  process,  the  IR&D  technical  evaluation  process  would  become  a 
cost-effective  means  of  managing  the  availability  of  future  technology. 

Navy 

How  is  the  IR&D  Technical  Evaluation  Process  Supposed  to  Work? 

Responsible  individuals  at  each  level  of  command  are  familiar  with  the 


17 


^^t^?SSST^s 


£*»  V*V-~>*-;  X^V»«'0 


:o 

i* 

V 


\tf~- 


process  as  described  in  appropriate  DOD  and  Navy  instructions. 

Annually  the  Navy,  as  lead  service,  evaluates  approximately  80  technical 
plans  and  conducts  approximately  30  on-site  reviews. 

How  Does  the  Process  Actually  Work? 

The  policies  described  in  DOD  Instruction  5100.66  are  implemented  by 
the  following  Naval  instructions: 

• SECNAV  Instruction  3900.40 

• Office  of  Naval  Research  Instruction  3900.32 

• Naval  Materiel  Command  Instruction  3900. 11 A 

SECNAV  Instruction  3900.40  designates  the  Assistant  Secretary  of  the 
Navy  (R&D)  and  the  Assistant  Secretary  of  the  Navy  (I&L)  as  Navy  repre- 
sentatives on  the  I R&D  Policy  Council.  In  addition,  the  Chief  of  Naval 
Research  is  designated  as  the  Navy  IR&D  Program  Manager  with  responsibility 
for  the  overall  conduct  of  IR&D  project  description  evaluations  and  on-site 
reviews  of  the  IR&D  programs. 

Office  of  Naval  Research  (0NR)  Instruction  3900.32  designates  the 
Director  of  Research  as  the  Departmental  IR&D  Manager,  and  therefore  the 
Navy's  member  on  the  DOD  IR&D  Technical  Evaluation  Group.  His  special 
assistant  for  IR&D  serves  as  alternate  member  of  the  Evaluation  Group  and 
as  the  overall  Navy  focal  point  for  coordination  of  all  IR&D  actions.  The 
instruction  also  assigns  the  following  responsibilities  to  ONR  Branch 
Offices  located  in  the  United  States: 

• Serve  as  lead  Navy  activity  for  designated  companies  and  conduct 
the  technical  ;valuation  of  their  IR&D  project  descriptions  and  on-site 
reviews. 

• Request  necessary  technical  assistance  from  other  Navy  activities 
for  each  IR&D  evaluation  and/or  on-site  review. 
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The  Navy  Materiel  Command's  (NMC)  participation  in  IR&D,  as  outlined 
in  Instruction  3900. 1 1 A,  is  to: 

• Require  active  participation  by  NMC  activities  in  the  technical 
evaluations  and  on-site  reviews  of  IR&D  programs. 

• Establish  the  NMC  IR&D  Technical  Manager. 

• Describe  the  responsibilities  of  NMC  activities  relative  to  the 
IR&D  program. 

Is  This  Method  Satisfactory?  If  Not,  What  Are  Some  Considerations 
That  Might  Improve  the  Process? 

The  procedures  contained  in  the  above  Naval  instructions  are  highly 
satisfactory;  and,  from  all  indications  those  procedures  are  being  complied 
with  at  all  levels  of  command.  There  is  unanimous  agreement  that  IR&D,  and 
hence,  the  technical  evaluation  process  is  highly  worthwhile;  however, 
there  is  difficulty  in  monitoring  the  process.  Like  the  Army,  the  Navy  is 
hampered  by  budget  and  personnel  constraints.  On-site  reviews  are  costly 
in  terms  of  per  diem  and  travel  funds.  One  attempt  at  overcoming  the 
shortage  of  funds  available  is  the  policy  of  assigning  ONR  Branch  Offices 
responsibility  for  respective  companies  on  a geographical  basis.  This 
method  does  not  always  provide  for  the  proper  break-out  of  qualified  personnel 
for  each  on-site  review. 

Additionally,  the  Navy  suffers  from  the  lack  of  personnel  dedicated  to 
IR&D  at  the  various  levels  of  command.  Specific  personnel  are  identified 
as  focal  points  and  address  IR&D  as  an  additional  duty,  or  on  an  as-necessary 
basis. 


Air  Force 

How  is  the  IR&D  Technical  Evaluation  Process  Supposed  to  Work? 
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All  individuals  interviewed  in  the  Air  Force  had  an  adequate  under- 
standing of  the  technical  evaluation  process  as  it  is  described  in  detail 
by  Air  Force  Regulations  and  documents.  Under  the  direction  of  the  Director 
of  Science  and  Technology  at  Headquarters  AFSC  effective  IR&D  evaluation 
procedures  have  been  identified  and  implemented.  The  DOD  Instruction  and 
Air  Force  Regulations  are  followed  quite  closely  by  Air  Force  Managers  in 
conducting  the  evaluation  of  the  Technical  Plan  and  in  on-site  reviews. 

In  1976,  91  technical  plan  evaluations  and  37  on-site  reviews  were 
conducted  by  the  Air  Force  as  lead  service.  In  addition,  the  Air  Force 
Technical  Manager  was  also  responsible  for  the  validation  of  24  technical 
plan  evaluations  and  10  on-site  reviews  done  by  the  Army  and  Navy  as  lead 
service  for  Air  Force  negotiated  companies  (17). 

How  Does  the  Process  Actually  Work? 

The  Mr  Force  follows  stated  guidelines  very  closely  in  conducting  the 
evaluations.  Specifically,  the  Technical  Manager  chairs  a meeting  in 
November  with  focal  points  from  Air  Force  Systems  Command  organizations 
that  are  parti ci pacing  in  the  evaluation.  The  focal  points  represent  all 
ten  Air  Force  Laboratories  plus  the  four  Product  Divisions  and  two  Centers. 
The  purpose  of  this  meeting  is  to  select  the  lead  organization  who  is 
responsible  for  organizing  the  evaluation  of  a particular  company.  By  mid- 
■january  the  on-site  review  schedule  is  established.  The  on-site  review  for 
a particular  company  now  takes  place  once  every  three  years;  thus,  a large 
portion  of  the  efforts  of  the  lead  laboratory  focal  point  is  involved  with 
the  annual  review  of  each  company's  technical  plan. 

The  lead  laboratory  focal  point  is  responsible  for  the  evaluation  of 
the  assigned  company.  He  must  interact  with  other  support  laboratory 
focal  points  to  identify  Air  Force  and  other  government  personnel  who  are 
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qualified  and  have  an  interest  in  participating  in  the  evaluation.  The 
lead  focal  point  collects  the  results  of  the  evaluation,  which  are  indicated 
on  the  DD  Form  1855,  and  prepares  the  DD  Form  1856  on  the  results  of  the 
evaluation.  These  documents  are  forwarded  to  the  Air  Force  Technical 
Manager  within  30  days  after  the  on-site  review,  and  within  120  days  after 
receipt  of  the  Technical  Plan.  The  technical  evaluations  are  an  important 
factor  in  the  negotiations  of  the  IR&D  ceilings  between  the  government  and 
the  company.  Changes,  up  or  down,  in  the  rating  of  the  company  from  one 
year  to  the  next  impact  the  negotiated  ceiling  according  to  a formula  used 
by  the  Directorate  of  Contract  Management  at  HQS  AFSC  (18).  Thus,  the 
current  company  effort  in  IR&D  is  compared  to  the  company's  effort  during 
the  previous  year,  and  not  with  the  effort  of  other  companies,  in  determining 
the  ceiling. 

An  important  benefit  of  the  evaluation  summary  report  is  the  feedback 
to  the  company  of  its  strong  and  weak  points  in  a timely  fashion.  Detailed 
results  of  the  evaluation  are  given  to  the  company  by  the  Air  Force  Technical 
Manager  with  names  of  organizations  and  individuals  deleted.  Necessary 
adjustments  in  the  IR&D  project  can  be  accomplished  after  receipt  of  this 
information. 

In  addition  to  this  feedback  by  the  Technical  Manager,  there  is  also  a 
de-brief  of  the  company's  on-site  IR&D  presentation  that  takes  place  before 
the  technical  evaluation  team  departs.  The  lead  service  focal  point  appoints 
from  among  the  evaluation  team  members  an  individual  to  serve  as  team 
leader  for  the  specific  scientific  area  being  presented  at  the  on-site. 

The  team  leader  focuses  in  on  specific  highlights  and  problem  areas  and 
presents  these  to  the  company  representatives  near  the  end  of  the  review. 

This  is  looked  upon  by  the  company  as  valuable  information  to  improve  their 


program. 


21 


n.,1^  J^. 


Many  of  the  Air  Force  managers  indicated  some  concern  regarding  the 
use  of  government  personnel  in  conducting  the  evaluation.  Their  concern 
was  in  the  qualifications  of  the  evaluator  and  whether  or  not  the  correct 
individuals  are  involved.  Evaluators  with  sufficient  knowledge  of  the  IR&D 
project  are  able  to  provide  higher  quality  evaluations.  Qualified  evalua- 
tors understand  the  details  of  the  project  and  are  able  to  provide  more 
critical  evaluations  and  can  provide  valuable  feedback  to  the  IR&D  project 
leader. 

Another  factor  here  is  the  idea  that  the  individual  evaluator  judges 
his  own  qualifications  and  indicates  this  in  block  7 of  the  DD  Form  1855. 
Numbers  are  assigned  to  each  qualification  level.  The  Air  Force  regulations 
require  a certain  minimum  number  for  the  sum  of  the  numerical  qualification 
factors  for  each  project  under  evaluation.  It  has  been  suggested  that  the 
evaluator's  supervisor  verify  the  qualification  level  indicated  to  provide 
more  objectivity  in  this  area. 

The  degree  of  participation  in  the  technical  evaluation  process  varies 
among  Air  Force  organizations  and  depends  strongly  upon  the  attitudes  of 
the  organization  commander.  His  active  encouragement  and  support  enhances 
the  probability  of  a successful  technical  evaluation. 

It  was  indicated  that  some  companies  identify  for  the  Technical  Manager 
9 or  focal  point  individuals  in  government  laboratories  who  the  company  feels 

can  provide  an  effective  evaluation.  One  Air  Force  Manager  indicated  that 
this  process  should  be  done  routinely.  It  would  seem  to  be  a good  idea  to 
establish  continuity  in  the  evaluation  if  the  same  qualified  individuals 
evaluated  the  annual  tech  plan  each  year  and  also  participated  in  the  on- 
site review. 

The  on-site  review  covers  only  a portion  of  the  total  IR&D  effort  of 
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the  company.  As  stated  previously,  only  30  percent  of  the  total  nurt  be 
reviewed.  Only  pre-selected  projects  are  presented  formally  at  the  on-site 
review.  Therefore,  in  order  to  obtain  an  idea  of  the  scope  of  the  IR&D 
effort,  the  on-site  evaluator  should  read  and  evaluate  beforehand  all  of 
the  projects  in  the  technical  plan  that  deal  with  the  scientific  area  under 
consideration. 

The  on-site  review  is  considered  by  most  managers  to  be  more  valuable 
than  the  readings  of  the  tech  plan  in  obtaining  information  and  providing 
judgments  concerning  the  quality  of  the  IR&D  project. 

In  addition  to  the  problems  of  identifying  the  qualified  evaluators 
and  gaining  their  interest  and  support,  there  is  also  the  factor  of  travel 
funds  required  to  attend  the  on-site  review.  As  far  as  is  known,  no  separate 
fund  citation  is  available  for  use  in  IR&D  travel.  Travel  for  IR&D  on-site 
reviews  competes  with  other  travel  requirements  for  funds.  This  is  perceived 
by  many  as  a definite  problem  in  terms  of  obtaining  sufficient  numbers  of 
qualified  individuals  to  attend  the  on-site  review. 

The  time  factor  also  plays  a role  in  the  quality  of  the  technical 
evaluation.  Many  highly  qualified  scientists  and  engineers  feel  a lack  of 
sufficient  time,  in  light  of  other  pressing  duties,  to  participate  in  the 
technical  evaluation.  Again,  the  attitude  of  the  organization  commander  is 
key  here.  If  he  places  strong  emphasis  on  the  need  for  effective  IR&D 
evaluations,  then  members  of  his  organizations  will  assign  a high  priority 
to  IR&D  technical  evaluation  participation.  However,  as  one  manager 
indicated,  the  involvement  by  scientists  must  be  strictly  voluntary;  they 
must  want  to  participate  in  order  to  be  effective. 

Is  This  Method  Satisfactory?  If  Not,  What  are  Some  Considerations  That 
Might  Improve  the  Process? 
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The  methods  and  procedures  that  the  Air  Force  uses  to  establish  quality 
level  and  potential  military  relevancy  of  IR&D  projects  is  by  no  means 
perfect.  However,  the  technical  evaluation  process  is  considered  by  all 
Air  force  managers  interviewed  to  be  effective  in  discharging  the  require- 
ments of  Section  203,  Public  Law  91-441.  As  stated  previously,  the  Air 
Force  has  established  effective  guidelines  through  the  issuance  of  general 
policy  statements  on  the  conduct  and  support  of  technical  evaluations  and 
in  assigning  responsibilities  and  describing  procedures  for  making  a technical 
evaluation  and  review  of  a contractor  IR&D  program.  Presently,  the  evalua- 
tion process  functions  effectively,  yet,  it  is  a dynamic  process.  Air 
Force  managers  are  continually  striving  to  improve  the  process. 

The  Air  Force  Audit  Agency  has  conducted  a recent  audit  of  the  technical 
evaluations.  This  audit  was  conducted  in  order  to  compare  actual  procedures 
with  standards  of  conduct  specified  in  Air  Force  Regulations.  This  evaluation 
has  uncovered  certain  discrepancies  in  complying  with  these  standards.  Most 
of  the  discrepancies  are  administrative  in  nature  and  can  probably  be 
corrected  by  establishing  tighter  control  of  these  factors. 

There  is  the  need  for  ensuring  maximum  project  evaluation  continuity 
where  emphasis  should  be  placed  on  consistency  in  the  year-to-year  evaluations 
by  providing  the  same  evaluators  for  the  technical  plan  and  for  the  on-site 
review.  The  organization  commander  would  play  a strong  role  in  providing 
for  this  continuity  by  ensuring  that  qualified  laboratory  scientific  personnel 
are  made  available,  as  required,  to  participate  in  the  evaluation  process. 

In  addition,  the  proper  balance  and  relationship  between  related  contractual 
R&D,  DOD  in-house  R&D,  and  IR&D  would  be  established  in  the  mind  of  the 
laboratory  scientist.  This  should  improve  utilization  of  the  IR&D  scientific 
information  by  the  government  and  provide  additional  benefits  to  the 


24 


Government  beyond  th' . primary  benefits  of  IR&D. 

Another  suggestion  to  improve  the  utilization  of  company  IR&D  was  to 
ensure  that  the  on-site  review  Lead  Organization  Summary  Report  is  provided 
to  the  AFPRO.  Since  the  APPRO  carries  out  the  interfacing  function  between 
the  company  and  the  Air  Force  Product  Division  where  systems  acquisition 
program  management  is  conducted,  IR&D  results  can  flow  to  the  Program 
Management  Office  in  a timely  manner.  AFSC,  DCS/Procurement  and  Manufacturing 
has  issued  a letter  to  HQ/AFCMD  providing  guidance  on  AFPRO  responsibilities 
in  surveillance  of  IR&D  and  B&P  efforts  (19). 

The  key  role  of  the  organization  IR&D  focal  point  is  self-evident.  As 
the  commander's  representative  his  ability  to  organize  an  effective,  quali- 
fied technical  evaluation  and  review  team  is,  perhaps,  the  most  important 
factor  in  the  quality  of  the  resulting  evaluation.  With  effective  backing 
by  his  commander  and,  perhaps,  with  a separate  funding  line  item  for  IR&D 
review  travel,  efficient  evaluations  can  be  obtained.  It  is  interesting  to 
note  that  during  Company  Fiscal  Year  (CFY)  1976  Air  Force  evaluation  organizations 
expended  23  manyears  at  a cost  of  $1  million  to  conduct  IR&D  evaluations. 

Of  the  dollar  total,  $100,000  was  for  per  diem  and  travel  (17). 

In  addition,  the  leadership  and  guidance  of  the  AFSC  Director  of 
Science  and  Technology  is  evident  in  the  Air  Force  technical  evaluation 
process.  An  example  of  this  guidance  is  found  in  a recent  policy  letter  sent 
to  Air  Force  organizations  that  participate  in  IR&D  evaluations  and  also  sent 
to  contractors.  This  letter  expressed  the  need  to  increase  emphasis  in  R&D 
focussed  toward  improvements  in  manufacturing  technology  in  order  to  find 
techniques  that  lead  to  reduction  in  acquisition  and  support  costs  of 
defense  systems.  IR&D  projects  directed  toward  manufacturing  areas  would 
be  accorded  equal  credit  to  projects  i other  technical  areas  during  Air 
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Force  technical  evaluations  (20). 


NASA 

Because  of  the  existence  of  common  areas  of  technological  interest, 

NASA  participates  extensively  with  the  Air  Force  in  IR&D  technical  evalua- 
tions. It  is  strongly  attuned  to  the  Air  Force  evaluation  procedures  and 
considers  them  to  be  effective  and  efficient.  It  was  indicated  that  NASA 
provided  a total  of  6 manyears  at  a cost  of  $200K  during  CFY  1976  for 
technical  evaluation  support  to  DOD.  This  is  roughly  20%  of  the  resources 
expended  by  the  Air  Force  in  conducting  its  technical  evaluations.  Several 
examples  were  cited  of  the  extensive  participation  and  support  provided  to 
Air  Force  lead  evaluations  by  NASA,  one  in  which  eight  NASA  scientists 
attended  an  on-site  review  to  support  a SAMSO  lead  organization  review  in 
1975.  One  of  the  authors  (JTV)  was  directly  involved  in  an  on-site  review  in 
which  four  NASA  scientists  attended  and  provided  important  input. 

NASA  views  participation  in  the  IR&D  technical  evaluation  process  as 
being  important  to  its  own  interests  because  it  provides  them  visibility 
into  the  entire  IR&D  program.  NASA  personnel  are  able  to  multiply  the 
scientific  knowledge  available  to  them,  thus  providing  them  with  the  leverage 
necessary  to  advance  the  state-of-the-art  in  their  respective  areas. 

Their  policy  is  to  expose  their  top  scientists  to  the  evaluation  and 
therefore  establish  a communication  link  between  the  company  ard  NASA  scien- 
tists. Dialogue  is  continued  after  the  evaluation  process  is  complete. 

Thus,  NASA  views  IR&D  technical  evaluation  as  an  effective  means  of  making 
industry  aware  of  its  needs.  DOD  would  also  benefit  by  viewing  IR&D  with 
this  attitude. 

The  NASA  manager  urges  the  government  scientists  to  pay  greater  attention 
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to  the  technical  content  of  the  IR&D  program.  He  encouraged  increased 
interaction  with  the  IR&D  project  leader  by  means  of  telephone  calls, 
visits,  and  briefings  after  the  on-site  reviews.  In  order  to  focus  on 
discrete  problem  areas,  round-table  discussions  were  advocated.  The  objective 
would  be  to  convey  to  industry  the  technical  needs  of  the  Government. 

Again,  it  was  considered  very  important  to  identify  the  right  people  and  to 
provide  encouragement  to  industry  scientists  but  to  avoid  providing  them 
direction  to  keep  the  R&D  independent. 

In  conclusion  the  technical  evaluation  process  works  and  it  should  be 
used  to  advantage. 


Industry 

It  is  evident  that  high  ranking  industry  u-t'f  '.ials  are  cognizant  of 
their  companies'  IR&D  program.  It  is  considered  an  important  part  of 
their  operations  because  it  provides  an  opportunity  to  develop  expertise 
and  experience  in  new  areas  of  technology  that  are  viewed  by  the  company  as 
potential  new  areas  of  business. 

According  to  one  company  IR&D  manager,  it  was  not  uncommon  for  members 
of  the  company  Board  of  Directors  to  participate  in  the  on-site  review. 

Typically,  an  on-site  review  begins  with  an  overview  briefing  by  a 
Division  Manager  that  describes  the  overall  company  policy  regarding  its 
planning  and  identification  of  responsiDility  for  its  IR&D  program.  Then 
an  assigned  Technical  Program  Manager  presents  a technical  description  of 
the  entire  program  followed  by  more  detailed  descriptions  by  project  leaders. 
There  is  generally  a good  question  and  answer  session  and  technical  inter- 
change in  a give-and-take  session. 

Most  companies  have  a positive  attitude  concerning  the  on-site  reviews. 
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One  company  representative,  in  commenting  on  the  evaluation  teams  from 
the  respective  services,  offered  the  following  information:  size  of  the 
on-site  review  teams  varied  from  as  few  as  ten  members  (Army  team)  to  as 
high  as  fifty-four  members  (Air  Force  team);  in  terms  of  which  projects 
were  to  be  reviewed  during  the  on-site  review,  the  Navy  indicates  to  the 
company  which  projects  they  wish  to  review;  the  Army  requests  the  company 
provide  a list  of  projects  that  they  would  like  reviewed;  and  the  Air  Force 
uses  a combination  of  these  techniques. 

In  addition,  this  company  representative  expressed  the  view  that  the 
Air  Force's  policy  of  reviewing  at  least  30  percent  of  the  IR&D  projects  at 
the  on-site  review  should  be  modified.  He  felt  it  is  more  beneficial  to 
examine  fewer  projects  in-depth  rather  than  examining  many  projects  superfi- 
cially. There  appears  to  be  no  consensus  as  to  the  relative  effectiveness 
of  the  three  Services'  methods.  Industry  believes  there  is  a serious 
effort  to  improve  the  quality  of  the  evaluators  by  calling  upon  top  scientists 
within  government  laboratories  to  participate  in  the  evaluations.  NASA  and 
the  Army  Missile  Research  and  Development  Command  were  cited  as  having  made 
considerable  progress  in  this  area.  The  Air  Force  Product  Divisions,  being 
more  management  oriented,  tended  to  experience  difficulty  in  finding  qualified 
technical  people  to  conduct  the  evaluations  when  they  assumed  lead  organiza- 
tion responsibilities.  The  situation  is  seen  to  be  improving. 

The  feedback  from  the  government  evaluation  team  is  considered  an 
important  input  for  considering  the  following  years'  IR&D  program.  Thus, 
it  was  emphasized  that  this  needed  to  be  sufficiently  detailed  and  provided 
in  a timely  fashion.  This  area  also  varied  considerably  from  service  to 
service.  The  Air  Force  gives  a detailed  debriefing  after  an  on-site  review; 
the  Army  gives  an  adequate  debriefing,  but  not  as  extensive  as  the  Air  Force; 
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and,  on  occasion,  the  Navy  has  not  provided  the  con.pany  being  evaluated 
any  feedback  whatsoever. 

It  would  seem  that  the  massive  exposure  received  by  evaluators  of  a 
company's  IR&D  program  would  lead  one  to  make  a strong  case  for  positive 
economic  benefits  that  offset  the  costs  required.  In  the  view  of  industry, 
knowledge  of  new  areas  of  technology  is  considered  very  valuable  and 
should  result  in  continuation  of  on-site  reviews. 
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SECTION  IV 


CONCLUSIONS  AND  RECOMMENDATIONS 
Benefits 

The  benefits  of  IR&D  programs  to  the  government  can  be  summarized  as 
follows: 

1.  They  provide  major  contributions  to  Nation's  technological  base 
and  avoidance  of  technical  surprises. 

2.  They  stimulate  competition  and  create  technical  alternatives  for 
government  requirements. 

3.  They  provide  cost-effective  technology. 

4.  They  provide  quick  reaction  and  flexibility  because  IR&D  work 
can  be  quickly  initiated,  terminated  or  redirected. 

5.  They  create  a climate  which  encourages  creativity. 

6.  They  reduce  risk  and  provide  responsiveness  to  weapons  acquisi- 
tion process. 

7.  They  generate  studies  and  system  concepts  supplementing  government 
planning. 

8.  They  enable  company  management  to  anticipate  and  respond  to 
changing  requirements. 

These  are  but  a few  examples  of  the  importance  and  value  of  IR&D  to 
the  government. 

All  IR&D  managers  interviewed  unanimously  agreed  that  the  evaluation 
process  provides  an  opportunity  for  open  communciation  between  the  govern- 
ment and  industry.  In  a different  context,  an  example  of  effective 
communication  follows. 
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When  a member  of  a Defense  Agency  asked  a Navy  R&D  manager  how  the 
Navy  transfers  in-house  R&D  results  to  the  User  Commands  and  Development 
Commands,  the  R&D  Manager  indicated  that  the  Navy  Laboratories  first 
communi cated  with  the  private  sector  by  taking  briefings  cn  R&D  results 
to  various  companies.  In  learning  what  R&D  the  Services  were  interested 
in,  industry  would  then  respond  more  effectively  by  planning  IR&D  that 
could  be  targeted  to  DOD  needs. 

This  example  appears  to  be  a circuitous  route,  but  it  is  known  to  be 
effective.  There  appears  to  be  no  doubt  that  open,  effective  communica- 
tion between  government  technical  personnel  and  industry  technical  person- 
nel can  be  catalyzed  by  the  IR&D  technical  evaluation  process.  This 
provides  a ready  opportunity  for  the  DOD  to  provide  feedback  and  encourage 
ment  to  industry  regarding  their  IR&D  efforts.  However,  industry  response 
to  DOD  comments  and  how  they  tailor  their  IR&D  program  must  be  carefully 
judged  in  order  to  avoid  excessive  direction  on  the  part  of  the  government 
which  would  cause  the  R&D  effort  to  become  dependent,  thus  unallowable. 

The  object  is  to  keep  the  word  "independent"  in  independent  research  and 
development. 

Thus,  the  DOD  must  strike  a balance  between  comprehensive  technical 
evaluations  and  determinations  of  PMR  as  required  by  law,  and  the  need  to 
permit  flexibility  and  innovation  in  IR&D.  Each  of  the  three  Services 
falls  at  a different  location  in  the  spectrum  of  the  technical  evaluation 
process  which  can  be  considered  to  range  from  a laissez  faire  attitude 
(hands-off  or  lenient)  at  one  extreme  to  a pragmatic,  task-oriented 
attitude  at  the  other.  Without  specifying  how  close  to  either  extreme 
any  one  Service  falls,  we  can  state  that  the  Navy  leans  toward  the  laissez 
faire  end  of  the  spectrum,  the  Air  Force  toward  the  pragmatic  end  with 
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the  Army  somewhere  between  the  other  two. 

Recommendations 

Recommended  areas  for  improvement  in  the  technical  evaluation  pro- 
cess follow. 

It  is  essential  that  qualified  government  individuals  participate  on 
a continuing  basis  in  che  technical  evaluation  process.  The  identification 
of  qualified  individuals  for  the  evaluation  team  is  an  area  that  requires 
some  consideration.  The  attempts  by  OSD  to  utilize  the  IR&D  data  bank 
and  the  DD  Form  1498  to  identify  common  technology  projects  and  the 
scientists  and  engineers  managing  these  projects  and  to  bring  these 
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people  together  should  be  continued.  Once  qualified  individuals  have 
oeen  identified,  it  is  recommended  that  they  continue  their  association 
with  the  evaluation  process  in  order  to  provide  continuity.  In  addition, 
the  same  evaluators  must  review  the  technical  plan  on  an  annual  basis  and 
participate  in  the  on-site  review.  One  method  of  accomplishing  this 
would  be  to  ensure  that  IR&D  Technical  Evaluation  would  be  part  of  an 
individual's  responsibilities  which  could  be  incorporated  into  his  job 
description.  It  is  also  recommended  that  a fund  cite,  strictly  for  use 
in  conducting  IR&D  evaluation,  should  be  created.  This  should  include 
funds  for  per  diem  and  travel  as  well  as  administrative  charges. 

For  the  benefit  of  the  evaluator,  it  is  recommended  that  an  IR&D 
Evaluator's  Handbook  be  generated  which  explains  the  IR&D  process,  the 
requirement  for  evaluations,  current  DOD  policy,  and  implementing  proce- 
dures for  conducting  a meaningful  evaluation.  This  would  be  prepared 
under  the  aegis  of  ODDR&E  with  guidance  from  the  IR&D  Policy  Guidance 
Council  and  the  Technical  Evaluation  Group  and  be  applicable  to  all  three 
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There  is  a feeling  that  too  few  individuals  in  the  weapons  acquisi- 
tion process  have  a proper  understanding  of  IR&D  and  its  value  to  the 
Government.  It  is  recommended  that  an  IR&D  position  paper  be  prepared, 
again  under  the  aegis  of  ODDR&E,  which  explains  in  some  detail  the  functions 
and  benefits  of  IR&D  and  to  encourage  the  involvement  of  development  and 
acquisition  personnel  in  the  technical  evaluation  process.  This  involvement 
need  not  be  for  the  purpose  of  technically  evaluating  the  IR&D  program 
but  to  define,  educate  and,  in  general,  explain  why  IR&D  is  important. 

In  general,  industry  appears  to  be  content  with  this  troika  of 
evaluation  procedures  which  they  encounter,  and  the  OSD  is  satisfied  that 
IR&D  is  accomplishing  what  it  is  intended  to  do  - foster  competition. 

As  stated  to  Congress  by  Dr.  Currie  in  1977: 

"We  believe  that  IR&D  is  a well-managed  cost  element  which 
contributes  to  the  competence  of  the  Defense  Industrial 
Base.  We  have  recently  taken  steps  to  provide  Congress 
with  an  overall  estimate  of  future  IR&D  expenditures.  This 
complements  our  previous  controls  and  provides  full  visibi- 
ty  to  permit  overall  policy  guidance  without  imposing  external 
direction  on  what  qualifications  a company  can  have.  Our 
success  in  Defense  research  and  development  depends  on 
industrial  competition,  and  that  competition  depends  on  the 
independence  of  IR&D."  (4 : IX-22 ) 

Natural  forces  of  evolution  will  continue  to  act  on  the  technical 
evaluation  process.  It  may  become  more  streamlined  or  more  efficient; 
however,  we  foresee  no  major  changes  in  the  present  method. 
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APPENDIX  A 


Persons  Interviewed 


Office  of  the  Secretary  of  Defense 
Mr.  James  W.  Roach 

Assistant  Director  (Engineering  Policy) 

Office  of  the  Director  of  Defense  Research  and  Engineering 
Chairman  of  the  IR&D  Technical  Evaluation  Group 

Department  of  the  Army 

Mr.  Charles  R.  Woodside 

Deputy  for  Management  and  Budget 

Office  of  the  Assistant  Secretary  of  the  Army  (R&D) 

Mr.  John  W.  Crellin 

Chief  of  Industry  Liaison  Office 

HQ,  Development  and  Readiness  Command 

IR&D  Departmental  Manager 

Department  of  the  Navy 

Ms.  Evelyn  Richards 
Staff  Assistant  (IR&D) 

Office  Chief  of  Naval  Research 
Alternate  IR&D  Departmental  Manager 

Department  of  the  Air  Force 

Major  Loren  Jacobsen 

DCS/Resoarch  and  Development  (AFRDPS) 

HQS  USAF 

Executive  Secretary  of  the  Air  Force  IR&D  Policy  Council 

Captain  William  Lewandowski 

Director  of  Science  and  Technology  (DLXB) 

HQS  Air  Force  Systems  Command 

Air  Force  IR&D  Technical  Manager/Departmental  Manager 
Mr.  Donn  V.  Aaby 

DCS/Procurement  and  Manufacturing  (PPMO) 

HQS  Air  Force  Systems  Command 
Chief,  Overhead  Management  Division 
Air  Force  Tri-Service  Negotiator,  Chief 

Captain  Hudson  Ratliff 
DCS/Procurement  and  Manufacturing  (PPMO) 

HQS  AFSC 
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Mr.  Matt  Kerper 

Air  Force  Office  of  Scientific  Research 
Air  Force  Systems  Command 
AFOSR  IR&D  Focal  Point 

NASA 


Dr.  Ralph  Nash 

NASA  Hqs,  Washington,  D.  C. 

Technical  Manager,  IR&D,  NASA  Headquarters 

Industry 

Elliot  B.  Harwood 

The  Boeing  Company 

Manager,  Corporate  IR&D  Activities 

D.  Max  Heller 
Martin-Marietta  Company 

Director  of  Research,  Martin-Marietta  Aerospace 
Walter  L.  Weitner 

Aerospace  Industries  Association  of  America 
Secretary,  CODSIA  IR&D/B&P  Task  Group 

James  F.  Lovett 

Westinghouse  Electric  Corporation 
Director,  Federal  Government  Relations 

William  Pearscn 

General  Electric  Company 

Corporate  IR&D  Representative 

Additional  comments  and  information  were  provided  by  the  following: 
Lt  Col  Richard  Hartke 

Office  of  the  Scientific  Advisor  to  the  President 

Lt  Col  Howard  Bethel 
DCS/Research  and  Development 
HQS  USAF 
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